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IMAGE SENSOR WITH MULTIPLE RESOLUTION AND COMPRESSION CAPABILITY 

Background of the Invention 

The present invention relates generally to an apparatus and method for 
interpolating and compressing image data and, more particularly, to an apparatus and 
method for sensing color sub-sampled image data and thereafter interpolating for 
non-color sub-sampled image data that substantially reduces color fringing and 
preserves detail while allowing selection of the overall quantity of image data. 

Electronic imaging cameras for recording still images are well known in the art. 
Such cameras can record a plurality of still images on a single magnetic disk or tape 
in either analog or digital format for subsequent playback on an output device, such 
as any well-known cathode ray tube (CRT) viewing device for example. Printers 
may also be utilized with such cameras in a well-known manner to provide hard 
copies of the recorded images. 

Such electronic imaging still cameras often utilize two-dimensional image 
sensing arrays such as charge coupled devices (CCDs) which integrate incident 
scene light over a predetermined time to provide an electronic information signal 
corresponding to the scene light intensity incident on the array. Such two- 
dimensional image sensing arrays comprise a predetermined number of discrete 
image sensing elements or pixels arranged in a two-dimensional airay, in which each 
image sensing element responds to incident illumination to provide an electronic 
information signal corresponding to the intensity of the incident of illumination. 
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In order to record color images, the incident illumination to the two dimensional 
image sensing array is filtered so that different image sensing elements receive 
different colored illumination. The filters are arranged in well known patterns across 
the face of an image sensing array, such as a repeating pattern of red, green, and blue 

5 stripes. Alternatively, individual image sensing elements or pixels across each line 
may be filtered in a repeating pattern of red, green, blue, and green filters in a two- 
by-two checkerboard, as is well known in the art. Since each image sensing element 
can only detect one color of illumination, the color information for the other colors 
not detected by that image sensing element must be filled in. Filling in the missing 

10 color information is generally accomplished by interpolating the detected image data 
for each color to determine color values for all the colors for each image sensing 
element. 

Conventional types of interpolation, however, can provide images with 
objectionable color aliasing artifacts such as "color fringes" near sharp edges. The 
15 conventional approach to solve this problem is to prevent the color fringes at the 
expense of image sharpness by blurring, otherwise known as anti-aliasing, the 
picture so that the edges are not sharp enough to create a color fringe. Blurring the 
image in this manner, however, has obvious disadvantages, resulting in a reduction 
in resolution and a so-called "fuzzy" picture. 

20 The missing color interpolation also increases the number of pixels by a factor of 

three from the raw image data sensed by the CCD. Storing the image data locally 
then oecomes a potential problem. The increased number of pixels means that a file 
size for storing the image, on local magnetic media for example, leads to a 
requirement for increased storage capacity. 
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The pixels on a CCD can be rectangular in shape particularly when striped color 
filter arrays are employed. Thus, outputting the image on an output device that has 
substantially square pixels distorts the image. Since most conventional output 
devices such as printers and CRT's require square pixels, it is necessary to form 
substantially square pixels from the rectangular pixels prior to outputting the image. 
A result is to get the desired pixels geometry. 

Summary 

The aforementioned and other objects are achieved by the invention which 
provides, in one aspect, an electronic still camera and a method associated therewith 
for image processing to form a processed image with reduced color artifacts, where 
an image is represented by a plurality of image signals in three colors. The method 
is useful with an electronic still camera, and comprises the steps of separating the 
plurality of image signals into a first color plane, a second color plane, and a third 
color plane. For example, many conventional CCDs use colored stripes formed onto 
the CCD to filter colors, thereby making individual pixels associated with only one 
color. These individual pixels are broken into color planes. Therefore, if colored 
stripes of red, green, and blue (RGB) are attached to the CCD, then the pixels 
associated with each individual color are broken into a red color plane, a green color 
plane, and a blue color plane. 

Since these colors have now been separated, voids now exist where the previous 
colors used to be. That is, in the red color plane, the pixel locations where green and 
blue had to have existed previously are now left void. Therefore, the next step is to 
interject interpolated image signals in each of the color planes that correspond to the 
image signal locations left void by the separation step. This now forms a triplet for 
each color location in the image. Previously, a single pixel location would contain 
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only information on one color, red for example. Now, with the interpolated data, 
each pixel location has RGB information associated therewith. 

Next, two difference signals are created at each pixel location that represent the 
differences between one channel and a second, and the first channel and the third 
5 within the RGB triplet. In the previous example using RGB, those difference signals 
can be any combination of the red, the green, or the blue, as long as from those three 
colors, two different difference signals are created. In the preferred embodiment, R- 
G and B-G are used. 

The difference signals are then filtered using a median filter which substantially 
10 reduces or removes color artifacts. The image is subsequently reconstructed from 
the original image signals and the filtered difference signals to form a processed 
image which has substantially preserved detail and reduced color artifacts. The 
improved RGB triplets thus formed from linearly interpolated RGB triplets have the 
same pixel geometry as the original single color pixels. In the case of rectangular 
15 pixel geometry, some method of producing pixels of square geometry must be 
employed for use with any display device that renders images on a raster of square 
geometry. 

In the case of rectangular pixels of aspect ratio height twice the width, the 
invention provides dual resolution modes for easily producing square pixels for 

20 display. In one resolution mode, neighboring pairs of pixels are averaged in the 
direction normal to the major axis of the rectangular pixels within each color plane 
such that one half as many square pixel RGB triplets are formed as there were 
original single color rectangular pixels, while in the second resolution mode new 
RGB triplets are interjected by interpolation in the direction parallel to the major 

25 axis of the rectangular pixels such that twice as many square pixel RGB triplets are 
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formed as there were original single color rectangular pixels. In the case of 
rectangular pixels of aspect ratio other than height twice width, multi resolution 
mode can be supported with more complicated down sampling and up sampling 
schemes. 

5 Finally, if the image data is to be stored, then compression becomes an essential 

characteristic. In the invention, the stored image signals are compressed versions of 
the two filtered difference signals and a compressed version of the single color 
signal common to both difference signals reconstructed from the filtered difference 
signals and the original sampled pixel values. In the preferred embodiment, 

10 compressed, filtered R-G and B-G are used as well as a compressed version of the G 
signal formed from the filtered R-G and B-G and the original RGB samples. The 
difference signals are highly compressible while the G signal formed from the 
filtered R-G and B-G and the original RGB samples provides the complete detail 
signal originally sensed, and the overall image size can be drastically compressed. 

15 Upon decompression of the compressed image data, pixels of square geometry can 
be reconstructed in either of the dual resolution modes. The aforementioned and 
other aspects of the invention are evident in the drawings and in the description that 
follows. 

Brief Description of the Drawings 

20 The foregoing and other objects of this invention, the various features thereof, 

as well as the invention itself, may be more fully understood from the following 
description, when read together with the accompanying drawings in which: 

Figs. 1A and IB show, respectively, pixel locations as found on a CCD, and a 
representative example; 
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Figs. 2A and 2B show the pixel locations of the example of Figs. 1A and IB, 
respectively, after linear interpolation is performed; 

Fig. 3 shows the representative example of Figs. IB and 2B during performance 
of median filtering; 

Fig. 4 shows the result of the median filter on the example of Fig. 3; 
Fig. 5 shows reconstruction of pixels of Fig. 3; and 

Figs. 6A, 6B ? and 6C show, respectively, the pixel values reconstructed after 
median filtration of the color difference signals, the pixel values after averaging in 
the direction normal to the major axis of the rectangular pixels, and the pixel values 
after interpolation along the major axis of the rectangular pixels. 

Detailed Description 

While the present invention retains utility within a wide variety of photographic 
devices, such as video cameras for example, and may be embodied in several 
different forms, it is advantageously employed in connection with an electronic still 
camera. Though this is the form of the preferred embodiment and will be described 
as such, this embodiment should be considered illustrative and not restrictive. 

As previously discussed, electronic imaging cameras conventionally record 
color images by using an image sensing array comprising a predetermined number of 
discrete image sensing elements or pixels arranged in a two-dimensional array in 
which the image sensing elements respond to incident illumination to provide an 
electronic information signal corresponding to the intensity of the incident 
illumination. Such an image sensing arrays may be a CCD of the frame transfer 
type. It is well known to sense color images using a single two-dimensional CCD 
array by filtering the illumination incident to the image sensing array so that 
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different groups of the image sensing elements arranged in well-known patterns 
across the image sensing array receive different wavelengths for colored 
illumination. Thus, each color of illumination is sampled by each group of image 
sensing elements, and thereafter interpolated to provide color values corresponding 
5 to the other groups of image sensing elements. The full color image is therefore 
estimated or interpolated between the different groups of image sensing elements or 
pixels to fill in all colors for each image sensing element or pixel. 

Conventional types of interpolation provide images with objectionable aliasing 
artifacts such as color fringes near sharp edges. An example of how a sharp edge in 
10 a sample to be recorded can create color fringes, when the image of the subject is 
reconstructed using conventional interpolation methods, will be discussed herein. 

As previously described, a typical CCD arrangement includes color filter stripes 
thereon such that individual stripes of pixels measure an intensity of light for only a 
single color. Figure 1A shows a typical arrangement of a CCD using color stripes, 
15 where individual pixels are marked by R, G, or B representing red, green, and blue 
colors. Numerical designations associated with the several colors, i.e., R] G| and 
Bi, together are referred to a triplet. 

In order to illustrate the invention, an example is shown for each step. The 
example is of an image having a sharp contrast such as a sharp white-to-black 

20 transition. The example begins in Fig. IB where a light intensity level is shown 
graphically for a white-to-black transition in an image. For the graph, the abscissa 
is i ; pixel number across the CCD and the ordinate is by signal level, often 
measured in volts. Keys to Fig. IB is as follows: the solid line represents the edge 
transition in the original continuous image of the scene before sampling, the circle 

>5 represents an original red sample, the triangle represents an original green sample, 

7 
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and the square represents an original blue sample. It can be seen from the 
illustration that for the original continuous image the falloff during the transition is 
substantially vertical. This indicates a sharp transition. A slope in the falloff would 
be indicative of a more gradual, or less sharp, transition. 

5 Fig. 2A again shows a pixel layout of the pixels of Fig. 1 A, but now the pixels 

have gone through the step of interpolation. In the preferred embodiment, this is a 
linear interpolation, as will be later hereinafter described. The convention used is 
that a capital letter represents original raw data and a lower case letter represents 
interpolated data. Therefore, the colors interpolated between R } and R 2 are shown as 

10 ri' and r,". Using linear interpolation, r,' contains 2/3 of the intensity of R,, and 1/3 
of the intensity of R 2 Likewise, ri" contains 1/3 the intensity of R, and 2/3 the 
intensity of R 2 . By linearly interpolating values throughout for red, green, and blue, 
the colors can be broken into three color planes such that there now exists triplets for 
each individual pixel location, where previously there only existed raw data in one 

15 color for each location. 

Continuing the example, Fig. 2B shows the step of linear interpolation on the 
white-to-black transition shown in Fig. IB. As shown in Fig. IB, the lines indicate 
paths of the linear interpolation where the circles, triangles, and squares represent 
original red, green, and blue samples and the x's represent interpolated values. 

20 Figure 2B illustrates how the interpolation tends to blur the sharp transition 
previously seen in the image by interpolating where the dashed line represents red, 
the dotted line represents green, and the solid line represents blue. The slope of the 
line is indicative of a less sharp transition, thus blurring the image. Fig. 2B also 
illustrates how the interpolation tends to create color fringe artifacts. In the original 

25 white-to-black transition the red, green, and blue intensity transitioned at the same 
spatial location while it can be seen in Fig. 2B that the red signal after interpolation 
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initiates transition before the green which initiates transition before the blue. An 
image reconstructed from the data represented in Fig. 2B would show a pronounced 
white-to-cyan-to-purple-to-blue-to-black color fringe. 

A next step is to create two difference signals at each location. The color fringe 
artifact at a white-to-black transition in the scene followed by interpolation within 
color planes would appear as a sudden rise in a color difference signal followed by a 
corresponding sudden fall or as a sudden fall followed by a sudden rise. It is this 
rapid increase and decrease in the difference between the colors which is a 
characteristic of objectionable color fringing, and is not simply a sudden rise in the 
difference between colors, which is indicative of a change from one color to a 
different color. Thus, it is unlikely that a real scene would result in the creation of 
such a color spike, and it is not desirable to create such a color spike as a result of a 
method of interpolation chosen. 

Fig. 3 illustrates the aforementioned color spikes, where in the preferred 
embodiment, the difference signals are performed by subtracting the intensity of 
green from the intensity of red and subtracting the intensity of green from the 
intensity of blue, that is R-G and B-G. One skilled in the art will realize that the 
actual choice of colors that determine the difference signals is somewhat arbitrary. 
Consistency after the choice is made is of primary importance. Therefore, other 
color difference signals can be chosen without detriment to the invention. Fig. 3(i) 
graphically illustrates the results of the difference signal for R-G, for the signal 
shown in Figure 2B. Fig. 3 (Hi) graphically illustrates B-G. 

Various methods can now be employed to remove the color aberration. In the 
preferred embodiment, a median filter is employed, such as that shown in commonly 
assigned patents 4,663,655, issued May 5, 1987, 4,774,565, issued September 27, 

9 
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1988, and 4,724,395, issued February 8, 1988, all to William T. Freeman, which are 
each incorporated herein by reference. Simply stated, the median filter takes a series 
of pixels, such as those shown in Fig. 3(ii), and replaces the pixel value at the center 
of the filter region with the median value of all the pixels within the region. Though 
5 various numbers of pixels can be used, in the preferred embodiment, the median 
filter uses nine pixels and returns the median value of the nine pixels. Therefore, as 
the filter is worked horizontally across the signal, it can be seen that unless the spike 
is longer than one half the pixel length, the spike in the signal will be essentially 
eliminated. 

10 Figure 4 illustrates the results of passing the median filter across the difference 

signals, where 4(i) corresponds to the R-G difference signal shown in 3(i), and 4(ii) 
corresponds to the B-G difference signal shown in 3(iii), where these signals can 
now be referred to as (R-G) 1 and (B-G) 1 , respectively. The differences are now flat 
and the reconstructed colors will now remain substantially constant with respect to 

15 each other before, during and after the color transition. Reference is again made to 
Figure IB as an example of this. 

A next step is shown in Figure 5, where the pixels are reconstructed from 
original raw RGB data and filtered (R-G)* and (B-G)' difference signals. The colors 
are restored using the following relationships: 

20 R@R = R 

R@G = G + (R-G) 1 
R@B = B - (B-G) 1 + (R-G)' 
G@R = R - (R-G) 1 
G@G = G 

10 
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G@B 



B - (B-G)' 



B@R 



R - (R-G)' + (B-G) 



B@G 



G + (B-G)' 



B@B 



10 



15 



20 



These relationships illustrate the properties of preserving the original sampled 
values and extracting detail at every pixel regardless of its original color. The detail 
extracting property is accomplished by removing the sudden spikes in the color 
difference signals which effectively produce local color sameness, so in regions of 
sharp edge transitions, when the original sampled pixels values change in response 
to edge transitions, the reconstructed missing color values are made to follow the 
original values. This property injects the detail information into all three channels. 
Figure 5 illustrates the reconstruction, where now instead of having diagonal lines 
showing the transition between colors, illustrating loss of sharpness and color 
fringing, now the circle representing red, the triangle representing green, and the 
square representing blue at the transition points are located in a substantially 
identical location such that the sharpness is retained and color fringes are avoided. 

In order to display the resulting image on most commercially available CRTs and 
most commercially available printers, a square pixel must now be attained. Since 
many CCDs, and specifically that of the preferred embodiment, have rectangular 
pixels, generally having a 2 to 1 aspect ratio, a way of solving this problem is to 
average two neighboring pixels, R, and rj\ for example, to form a single square 
pixel at that location. Fig. 6A shows pixels values from an RGB striped sensor with 
2:1 aspect ratio rectangular pixels after reconstruction from original sampled color 
values and median filtered color difference signals. Figure 6B shows pixel values on 
a square grid after averaging neighboring values. This would take an image 



n 
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resolution that has one million pixels, for example, and create an image that is 
displayable to a user having 500,000 three color pixels. An alternative method of 
creating square pixels is optionally used in the preferred embodiment. Rather than 
averaging horizontally, the pixels are interpolated vertically, as is shown in Fig. 6C. 

5 In Fig. 6C(i), an original line of RGB data is shown, and in Fig. 6C(iii) a second 

line of RGB data is shown, separated by Fig. 6C(ii) which is an interpolated line of 
RGB data. The interpolation between these lines can be as simple as nearest 
neighbor interpolation, which, in essence, duplicates a previous line to create a new 
line. The preferred methods include linear interpolation, bi-cubic convolution 
10 interpolation, or frequency domain interpolation, such as Fourier, DCT, wavelet, et 
cetera. 

The interpolated line of Fig. 6C(ii) creates additional pixels available for display. 
Thus, in the one million pixel example, two million three color pixels would now be 
available. That is, if the image was 1600 * 600 * 3 where the three represents the 
15 three color planes, interpolation would result in 1600 * 1200 * 3 available pixels. 

In contrast to the prior art method of beginning and ending with square pixels, 
the method of the invention does not, per se, result in square pixels. The pixel 
geometry has actually not changed. The method works since output devices do not 
actually know the geometry of the pixels coming into the device, but instead simply 
20 displays according to an electronic signal. Therefore, rectangular pixels are 
displayed either separated by an interpolated line which carries color information 
that makes the transition between the rectangular pixels or effectively formed into 
squares by averaging. This serves to eliminate the distortion that would otherwise 
result and simulates having a square pixels geometry. 

12 
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The resulting increase in pixels bodes well for output image quality but presents 
a difficulty for data storage. Increasing the image pixel count by two requires an 
increase in storage capacity for an image by two. Therefore, compression is used to 
decrease image storage requirements. 

5 A characteristic that makes data highly compressible is redundancy. The actual 

amount of compression is determined by the compression method chosen. In the 
preferred embodiment, DCT compression is used which is described in commonly 
assigned patent application number 08/441,000 filed May 15, 1995 which is 
incorporated herein by reference. 

0 Signals that have a high degree of redundancy are the filtered difference signals 

as can be seen in Figure 4. The signals are compressed along with the reconstructed 
data of one color. The one color must be chosen such that the one color can be used 
to derive the remaining colors with the difference signals. In this case, green is used 
since both difference signals incorporate green. Therefore, the other colors can be 

5 derived by the following equations where the superscripted plus indicates that 
reconstructed data is incorporated therein: 

G = G + 

R = G + + (R-G)' 
B = G + + (B-G) 1 

0 The detail is carried in the green G + signal which when the image is 

reconstructed is injected into the R and B signals by the above equations. 

In the preferred embodiment, visually lossless compression is used though 
lossless compression can be used without detriment to the invention. Visually 
lossless is preferred since higher compression ratios can be obtained, on the order of 

13 
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10:1, while by definition changes to the reconstructed image are not detected to the 
human eye. Furthermore, in the preferred embodiment, generation of RGB pixel 
values with square geometry in either of two resolutions is accomplished during 
decompression as described in commonly assigned patent application number 
5 08/441,000. 

The invention may be embodied in other specific forms without departing from 
the spirit or essential characteristics thereof. The present embodiments are, 
therefore, to be considered in all respects as illustrative and not restrictive, the scope 
of the invention being indicated by the appended claims rather than by the foregoing 
10 description, and all changes which come within the meaning and range of 
equivalency of the claims are therefore intended to be embraced therein. 



14 
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Claims 

The embodiments of the invention in which an exclusive property or privilege is 
claimed are defined as follows: 

1 . A method of processing an image stored in an electronic still camera in a first 
resolution having a first pixel geometry and the electronic still camera is in 
communication with a display device having a second pixel geometry, the 
method comprising: 

5 processing the image to alter the first pixel geometry to form a processed 

image having the second pixel geometry; and 

transmitting the processed image to the display device for display thereon; 

2. The method according to claim 1 wherein said step of processing further 
comprises the steps of: 

separating the plurality of image signals into a first color plane, a second 
color plane and a third color plane; 

5 interjecting interpolated image signals into each color plane corresponding to 

image signal locations left void by said separation step to form a 
triplet for each location in the image; 

creating a first difference signal and a second difference signal representing a 
difference between two or more image signals for each triplet; 

10 filtering the first and second difference signals to create filtered difference 

signals; and 

reconstructing the image from the plurality of image signals and said filtered 
difference signals to form the processed image having the second 
pixel geometry. 

3. A method according to claim 2 wherein said step of reconstructing the image 
comprises generating pixel values according to the following criteria: 

R@R=R; R@G=G+(R-G)'; R@B=B-(B-G)'+(R-Gy 

where (B-G)' and (R-G)' are the first and second difference signals, 
5 respectively. 



15 
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4. The method according to claim 2 further comprising the step of reducing a 
file size of the processed image for storage by compressing said difference 
signals and one color common to the difference signals. 

5. The method according to claim 1 further comprising the steps of: 

receiving a resolution signal from the display device; and 

altering the image such that the processed image has a second resolution not 
equal to the first resolution. 

6. The method according to claim 5 wherein said step of altering the image 
comprises the step of averaging pixels in the image horizontally to form the 
processed image. 

7. The method according to claim 4 wherein said step of altering the image 
comprises the step of interpolating pixels in the image vertically to form the 
processed image. 

8. The method according to claim 7 wherein said step of interpolating pixels in 
the image uses linear interpolation. 

9. The method according to claim 7 wherein said step of interpolating pixels in 
the image uses bi-cubic convolution interpolation 

10. The method according to claim 7 wherein said step of interpolating pixels in 
the image uses frequency domain interpolation 

1 1 . The method according to claim 7 wherein said step of interpolating pixels in 
the image uses nearest neighbor interpolation. 

12. An electronic still camera having a sensor with rectangular pixels which 
captures image-bearing light to form an image thereon, said camera being 
connectable to a computer having a display with square pixels, said camera 
further comprising processor means for converting rectangular pixels in the 
image to produce square pixels such that the image can be displayed on the 
display of the computer. 

13. The electronic still camera according to claim 12 wherein said processing 
means is selectable to transmit the image to the computer in one of a plurality 
of resolutions. 
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14. The electronic still camera according to claim 12 wherein said processing 
means is selectable to transmit the image to the computer in a first resolution 
or a second resolution mode where the first resolution mode is less than the 
second resolution mode. 

15. The electronic still camera according to claim 12 wherein said processing 
means averages neighboring pairs of the rectangular pixels in the direction 
normal to a major axis of the rectangular pixels to form the first resolution 
mode. 

16. The electronic still camera according to claim 12 wherein half as many 
square pixel RGB triplets are formed as there were original single color 
rectangular pixels. 

17. The electronic still camera according to claim 12 wherein said processing 
means interjects interpolated pixels in the direction parallel to the major axis 
of the rectangular pixels to form the second resolution mode. 

18. The electronic still camera according to claim 12 wherein twice as many 
square pixel RGB triplets are formed as there were original single color 
rectangular pixels. 

19. The electronic still camera according to claim 12 wherein the image is 
compressed prior to storage. 

20. A method of storing an image captured by an electronic imaging device 
comprising the steps of: 

separating the image into a first color plane, a second color plane and a third 
color plane; 

interjecting interpolated pixels into each color plane corresponding to image 
signal locations left void by the separation step to form a triplet for 
each location in the image to produce complete color planes; 

creating a first difference signal and a second difference signal representing a 
difference between two or more image signals for each triplet; 

compressing the first difference signal, the second difference signal and one 
of the complete color planes where the one of the complete color 
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planes is chosen as being a color common to the first and second 
difference signals. 

2 1 . The method according to claim 20 further comprising the step of 
decompressing the first difference signal, the second difference signal and 
one of the complete color planes to form a decompressed image substantially 
similar to the image. 

22. The method according to claim 20 wherein said step of decompressing 
performs reconstruction according to the following formula where G is 
representative of the one of the complete color planes common to the first 
and second difference signals: 

G = G + 

R = G + + (R-G)' 
B = G + + (B-G)' 

and where the superscripted plus indicates that reconstructed data is 
incorporated therein. 

23. The method according to claim 22 wherein the G + signal contains color detail 
which when reconstructed is injected into the R and B signals by the above 
equations. 
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